Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws-sdk-php tests fail for server/gateway #216

Closed
nitisht opened this issue Nov 14, 2017 · 4 comments · Fixed by minio/minio#5188
Closed

aws-sdk-php tests fail for server/gateway #216

nitisht opened this issue Nov 14, 2017 · 4 comments · Fixed by minio/minio#5188
Assignees
Milestone

Comments

@nitisht
Copy link
Contributor

nitisht commented Nov 14, 2017

Mint test on latest rc failed. Reproducible with both Server and Gateway modes

Mint logs:

To get logs, run 'sudo docker cp 7bf35828e9e5:/mint/log /tmp/mint-logs'
(1/9) Running aws-sdk-php tests ... FAILED in 1 seconds
{
  "name": "aws-sdk-php",
  "function": "presignedPut ( array $params = [] )",
  "args": {
    "Bucket": "aws-sdk-php-5a0a92dc59ad6",
    "Object": "obj1"
  },
  "duration": "13",
  "status": "FAIL",
  "error": "Server error: `PUT http://127.0.0.1:9000/aws-sdk-php-5a0a92dc59ad6/obj1?X-Amz-Content-Sha256=invalid-sha256&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20171114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171114T065317Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1200&X-Amz-Signature=6f0bc30857ea246de1d84b92d91238d9aa83b21168727017e844b820e6a523c8` resulted in a `500 Internal Server Error` response:\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, pleas (truncated...)\n"
}

Server logs:

root@nitish-mint-test:~# ./minio server /tmp/test
Endpoint:  http://147.75.94.237:9000  http://10.64.0.129:9000  http://172.17.0.1:9000  http://127.0.0.1:9000
AccessKey: minio 
SecretKey: minio123 
Region:    us-east-1

Browser Access:
   http://147.75.94.237:9000  http://10.64.0.129:9000  http://172.17.0.1:9000  http://127.0.0.1:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://147.75.94.237:9000 minio minio123

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide

Drive Capacity: 99 GiB Free, 102 GiB Total
ERRO[0007] Unable to fetch bucket info.                  cause=Bucket name invalid: aws-sdk-php-5a0a92dc59ad6-- source=[bucket-handlers.go:130:objectAPIHandlers.GetBucketLocationHandler()] stack=fs-v1.go:193:fsObjects.getBucketDir fs-v1.go:201:fsObjects.statBucketDir fs-v1.go:229:fsObjects.GetBucketInfo <autogenerated>:1:(*fsObjects).GetBucketInfo bucket-handlers.go:129:objectAPIHandlers.GetBucketLocationHandler api-router.go:69:(objectAPIHandlers).GetBucketLocationHandler-fm
ERRO[0007] Unable to fetch bucket info.                  cause=Bucket name invalid: aws-sdk-php-5a0a92dc59b12-- source=[bucket-handlers.go:659:objectAPIHandlers.HeadBucketHandler()] stack=fs-v1.go:193:fsObjects.getBucketDir fs-v1.go:201:fsObjects.statBucketDir fs-v1.go:229:fsObjects.GetBucketInfo <autogenerated>:1:(*fsObjects).GetBucketInfo bucket-handlers.go:658:objectAPIHandlers.HeadBucketHandler api-router.go:89:(objectAPIHandlers).HeadBucketHandler-fm
ERRO[0007] Unable to delete object. obj1-copy0           cause=Prefix access is denied: aws-sdk-php-5a0a92dc59ad6/obj1-copy0 source=[bucket-handlers.go:337:objectAPIHandlers.DeleteMultipleObjectsHandler()]
ERRO[0007] Unable to delete object. obj1-copy1           cause=Prefix access is denied: aws-sdk-php-5a0a92dc59ad6/obj1-copy1 source=[bucket-handlers.go:337:objectAPIHandlers.DeleteMultipleObjectsHandler()]
ERRO[0007] Unable to delete object. obj1-copy2           cause=Prefix access is denied: aws-sdk-php-5a0a92dc59ad6/obj1-copy2 source=[bucket-handlers.go:337:objectAPIHandlers.DeleteMultipleObjectsHandler()]
ERRO[0008] Unable to complete multipart upload.          cause=Part size for 1 should be at least 5MB source=[object-handlers.go:1061:objectAPIHandlers.CompleteMultipartUploadHandler()] stack=fs-v1-multipart.go:806:fsObjects.CompleteMultipartUpload <autogenerated>:1:(*fsObjects).CompleteMultipartUpload object-handlers.go:1059:objectAPIHandlers.CompleteMultipartUploadHandler api-router.go:52:(objectAPIHandlers).CompleteMultipartUploadHandler-fm

@nitisht nitisht added this to the Current milestone Nov 14, 2017
@krisis
Copy link
Member

krisis commented Nov 14, 2017

Findings from investigation

  1. aws-sdk-php doesn't include x-amz-contentsha256 in signed headers
http://s3.amazonaws.com/aws-sdk-php-5a0aeaaedaad0/obj1?X-Amz-Content-Sha256=invalid-sha256&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=**REDACTED**%2F20171114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171114T130808Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1200&X-Amz-Signature=**REDACTED**
  1. Presigned PUT URL with invalid ContentSha256 completes successfully on AWS S3, i.e sha256 sum is not validated in a presigned PUT request. This can be verified using the following sample php program.
<?php
require 'vendor/autoload.php';

use Aws\S3\S3Client;
use Aws\Credentials;
use Aws\Exception\AwsException;
use GuzzleHttp\Psr7;
use GuzzleHttp\Client;

$s3Client = new S3Client([
    'credentials' => [
        'key' => 'accessKey',
        'secret' => 'secretKey',
    ],
    'scheme' => 'http',
    'use_path_style_endpoint' => true,
    'region' => 'us-east-1',
    'version' => '2006-03-01'
]);
$bucket = 'mybucket';
$object = 'myobject';
$sha256 = "invalid-sha256";

$cmd = $s3Client->getCommand('PutObject', [
        'Bucket' => $bucket,
        'Key'    => $object,
        'ContentSHA256' => $sha256,
]);

$request = $s3Client->createPresignedRequest($cmd, '+20 minutes');
$presignedUrl = (string) $request->getUri();
echo $presignedUrl;

$client = new Client(['debug' => true]);
$body = "Hello, World";
$res = $client->request('PUT', $presignedUrl, ['body' => $body]);
echo 'statusCode: ' . $res->getStatusCode();

?>

Resolution

  1. aws-sdk-php mint test
    This test doesn't capture a valid use-case, nor is it a negative test-case. ContentSha256 is ignored in a presigned PUT request. To quote from AWS S3 docs,

You don't include a payload hash in the Canonical Request, because when you create a presigned URL, you don't know the payload content because the URL is used to upload an arbitrary payload. Instead, you use a constant string UNSIGNED-PAYLOAD.

  1. minio-server
    Minio server should not validate content-sha256 for presigned requests.

@harshavardhana
Copy link
Member

We should change the test to incorporate this perhaps

package main() {
      //creds := credentials.NewStaticCredentials("USWUXHGYZQYFYFFIT3RE", "MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03", "")                                                                                          
        creds := credentials.NewStaticCredentials(os.Getenv("ACCESS_KEY"), os.Getenv("SECRET_KEY"), "")                                                                                                              
        newSession := session.New()                                                                                                                                                                                  
        s3Config := &aws.Config{                                                                                                                                                                                     
                Credentials: creds,                                                                                                                                                                                  
                //Endpoint:         aws.String("http://localhost:9000"),                                                                                                                                             
                Endpoint: aws.String("https://s3.amazonaws.com"),                                                                                                                                                    
                Region:   aws.String("us-east-1"),                                                                                                                                                                   
                //DisableSSL:       aws.Bool(true),                                                                                                                                                                  
                //S3ForcePathStyle: aws.Bool(true),                                                                                                                                                                  
        }                                                                                                                                                                                                            
                                                                                                                                                                                                                     
        // Create an S3 service object in the default region.                                                                                                                                                        
        s3Client := s3.New(newSession, s3Config)                                                                                                                                                                     
        req, _ := s3Client.PutObjectRequest(&s3.PutObjectInput{                                                                                                                                                      
                Bucket:      aws.String("vadmeste"),                                                                                                                                                                 
                Key:         aws.String("object"),                                                                                                                                                                   
                ContentType: aws.String("application/octet-stream"),                                                                                                                                                 
        })                                                                                                                                                                                                           
                                                                                                                                                                                                                     
        req.HTTPRequest.Header.Set("Content-Md5", "82bb413746aee42f89dea2b59614f9ef")                                                                                                                                
        url, err := req.Presign(24 * time.Hour)                                                                                                                                                                      
        if err != nil {                                                                                                                                                                                              
                panic(err)                                                                                                                                                                                           
        }                                                                                                                                                                                                            
                                                                                                                                                                                                                     
        rreq, err := http.NewRequest("PUT", url, bytes.NewReader([]byte("")))                                                                                                                                        
        rreq.Header.Add("Content-Type", "application/octet-stream")                                                                                                                                                  
        rreq.Header.Set("Content-Md5", "82bb413746aee42f89dea2b59614f9ef")                                                                                                                                           
        log.Println("REQUEST PUT ", url)                                                                                                                                                                             
        for k, v := range rreq.Header {                                                                                                                                                                              
                log.Printf("%v: %v", k, v)                                                                                                                                                                           
        }                                                                                                                                                                                                            
                                                                                                                                                                                                                     
        resp, err := http.DefaultClient.Do(rreq)                                                                                                                                                                     
        if err != nil {                                                                                                                                                                                              
                panic(err)                                                                                                                                                                                           
        }                                                                                                                                                                                                            
        body, err := ioutil.ReadAll(resp.Body)                                                                                                                                                                       
        resp.Body.Close()                                                                                                                                                                                            
        if err != nil {                                                                                                                                                                                              
                panic(err)                                                                                                                                                                                           
        }                                                                                                                                                                                                            
        log.Println("AMZ RESPONSE")                                                                                                                                                                                  
        for k, v := range resp.Header {                                                                                                                                                                              
                log.Printf("%v: %v", k, v)                                                                                                                                                                           
        }                                                                                                                                                                                                            
        log.Printf("\n%v", string(body))      
}                          
$ go run put-object-presign-content-type.go
2017/11/14 10:14:17 REQUEST PUT  https://vadmeste.s3.amazonaws.com/object?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJY2EP7JTV2SUOH4Q%2F20171114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171114T181417Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost&X-Amz-Signature=1c1dec7ce9531106bd437d8850589182ff9ae3b561d654b43133a732130d73e4
2017/11/14 10:14:17 Content-Md5: [82bb413746aee42f89dea2b59614f9ef]
2017/11/14 10:14:17 Content-Type: [application/octet-stream]
2017/11/14 10:14:18 AMZ RESPONSE
2017/11/14 10:14:18 Server: [AmazonS3]
2017/11/14 10:14:18 X-Amz-Request-Id: [2F8B8FA82C2BBECB]
2017/11/14 10:14:18 X-Amz-Id-2: [C8t+C+nOXAaALT+uFgarF6q+ALJJUP8Xz1l0PPCK92vtHWdZRvZd+dwGzd5DZcywS24DTNVwyg0=]
2017/11/14 10:14:18 Content-Type: [application/xml]
2017/11/14 10:14:18 Date: [Tue, 14 Nov 2017 18:14:18 GMT]
2017/11/14 10:14:18 
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidDigest</Code><Message>The Content-MD5 you specified was invalid.</Message><Content-MD5>82bb413746aee42f89dea2b59614f9ef</Content-MD5><RequestId>2F8B8FA82C2BBECB</RequestId><HostId>C8t+C+nOXAaALT+uFgarF6q+ALJJUP8Xz1l0PPCK92vtHWdZRvZd+dwGzd5DZcywS24DTNVwyg0=</HostId></Error>

@krisis
Copy link
Member

krisis commented Nov 15, 2017

Resolution (Updated)

  • aws-sdk-php mint test
    This test does capture a valid yet uncommon use-case where the presigned URL provider wishes to control the content to match the given sha256 sum. But aws-sdk-php doesn't include the x-amz-content-sha256 in signed headers. i.e, not including X-Amz-Content-Sha256 in presigned request header doesn't fail with SignatureDoesNotMatch. I.e, when this test is run on Minio server/gateway it would return different error code than when run on AWS S3. So, it makes sense to add an equivalent test using aws-sdk-go and remove this test.

  • minio-server
    Minio server should return ErrContentSHA256Mismatch if an invalid hex value is set in X-Amz-Content-Sha256 header.

@nitisht
Copy link
Contributor Author

nitisht commented Nov 17, 2017

This PR #222 is needed as well

@nitisht nitisht reopened this Nov 17, 2017
@nitisht nitisht added the fixed label Nov 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants