New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No way to enforce V2 / ceph #372

Closed
bornmw opened this Issue Mar 5, 2015 · 9 comments

Comments

Projects
None yet
6 participants
@bornmw

bornmw commented Mar 5, 2015

There is ENFORCE_S3_SIGV4_SYSTEM_PROPERTY that enforces V4 signatures.
But there is no way to enforce V2 signatures.

By default S3 SDK uses V4 for "non-standard" endpoints. This practically makes it impossible to use S3 SDK to talk to Ceph since Ceph doesn't support v4 signatures.

It seems like a straightforward fix.

@manikandanrs

This comment has been minimized.

Show comment
Hide comment
@manikandanrs

manikandanrs Mar 5, 2015

Contributor

Not all regions support v2 signatures. SDK uses this configuration to decide on the signature version to be used and by default it uses v4.

However for GetObject requests that uses a non standard endpoint, SDK upgrade's to v4. This is because requests that fetch AWS KMS encrypted objects needs to be v4 signed and there is no way SDK can now differentiate a request that fetches a AWS KMS encrypted objects and other objects.

Contributor

manikandanrs commented Mar 5, 2015

Not all regions support v2 signatures. SDK uses this configuration to decide on the signature version to be used and by default it uses v4.

However for GetObject requests that uses a non standard endpoint, SDK upgrade's to v4. This is because requests that fetch AWS KMS encrypted objects needs to be v4 signed and there is no way SDK can now differentiate a request that fetches a AWS KMS encrypted objects and other objects.

@bornmw

This comment has been minimized.

Show comment
Hide comment
@bornmw

bornmw Mar 5, 2015

All that is valid and understood. But makes it incompatible with Ceph, and probably some other use cases. Do you think there's any possibility to add a flag to enforce V2?

bornmw commented Mar 5, 2015

All that is valid and understood. But makes it incompatible with Ceph, and probably some other use cases. Do you think there's any possibility to add a flag to enforce V2?

@hansonchar

This comment has been minimized.

Show comment
Hide comment
@hansonchar

hansonchar Mar 5, 2015

Contributor

IIRC, there is a programmatic way to override the signer algorithm via ClientConfiguration:

    ClientConfiguration.setSignerOverride(final String value);

For example, you can do something like:

    ClientConfiguration.setSignerOverride("S3SignerType");
Contributor

hansonchar commented Mar 5, 2015

IIRC, there is a programmatic way to override the signer algorithm via ClientConfiguration:

    ClientConfiguration.setSignerOverride(final String value);

For example, you can do something like:

    ClientConfiguration.setSignerOverride("S3SignerType");
@bornmw

This comment has been minimized.

Show comment
Hide comment
@bornmw

bornmw Mar 6, 2015

Unfortunately setSignerOverride doesn't override anything in case of a non-standard endpoint and GET request :(

com.amazonaws.services.s3.AmazonS3Client.createSigner gets signer first (that's when signerOverride comes into play) but then since it is non-standard endpoint it returns v4Signer:

    Signer signer = getSigner();
    ...
    final boolean isStandardEnpoint = getEndpoint().getHost().equals(
            Constants.S3_HOSTNAME);
    ...
    final boolean sigv4ForGetRequests = ((originalRequest instanceof GetObjectRequest) && !isStandardEnpoint);
    ...
    if (
            !(signer instanceof AWSS3V4Signer) &&
            (upgradeToSigV4() || sigv4ForGetRequests)
       ) {
            ...
            return v4Signer;

(it returns in the middle of method body)

Also, (upgradeToSigV4() || sigv4ForGetRequests) calls upgradeToSigV4 method on each GET request irrespective of sigv4ForGetRequests, it would be nice to change it to (sigv4ForGetRequests || upgradeToSigV4()) to skip method call to save some cpu cycles.

bornmw commented Mar 6, 2015

Unfortunately setSignerOverride doesn't override anything in case of a non-standard endpoint and GET request :(

com.amazonaws.services.s3.AmazonS3Client.createSigner gets signer first (that's when signerOverride comes into play) but then since it is non-standard endpoint it returns v4Signer:

    Signer signer = getSigner();
    ...
    final boolean isStandardEnpoint = getEndpoint().getHost().equals(
            Constants.S3_HOSTNAME);
    ...
    final boolean sigv4ForGetRequests = ((originalRequest instanceof GetObjectRequest) && !isStandardEnpoint);
    ...
    if (
            !(signer instanceof AWSS3V4Signer) &&
            (upgradeToSigV4() || sigv4ForGetRequests)
       ) {
            ...
            return v4Signer;

(it returns in the middle of method body)

Also, (upgradeToSigV4() || sigv4ForGetRequests) calls upgradeToSigV4 method on each GET request irrespective of sigv4ForGetRequests, it would be nice to change it to (sigv4ForGetRequests || upgradeToSigV4()) to skip method call to save some cpu cycles.

@manikandanrs

This comment has been minimized.

Show comment
Hide comment
@manikandanrs

manikandanrs Mar 6, 2015

Contributor

@bornmw

A couple of questions

  1. What is the endpoint that you are using ?
  2. What SDK version are you using ? The latest SDK first derives the signer based on SignerOverride and the endpoint. It skips the upgrade if the signerOverride is set by the user.

The code that you have pasted above is from an older version of the SDK. Can you try upgrading to the latest version ?

Contributor

manikandanrs commented Mar 6, 2015

@bornmw

A couple of questions

  1. What is the endpoint that you are using ?
  2. What SDK version are you using ? The latest SDK first derives the signer based on SignerOverride and the endpoint. It skips the upgrade if the signerOverride is set by the user.

The code that you have pasted above is from an older version of the SDK. Can you try upgrading to the latest version ?

@bornmw

This comment has been minimized.

Show comment
Hide comment
@bornmw

bornmw Mar 6, 2015

I was on 1.9.9.
Switched to 1.9.23 and it has that logic rewritten, fixing the issue.

Thanks!

bornmw commented Mar 6, 2015

I was on 1.9.9.
Switched to 1.9.23 and it has that logic rewritten, fixing the issue.

Thanks!

@bornmw bornmw closed this Mar 6, 2015

@jamshid

This comment has been minimized.

Show comment
Hide comment
@jamshid

jamshid Sep 3, 2015

So to sum up, the way to make aws-sdk-jar 1.10.15 work with an S3-compatible service that does not implement V4 signatures is to:

        ClientConfiguration opts = new ClientConfiguration();
        opts.setSignerOverride("S3SignerType");  // NOT "AWS3SignerType"
        AmazonS3Client s3 = new AmazonS3Client(opts);
        s3.setEndpoint("https://storage.example.com");

It's tricky because as mentioned earlier, the SDK really wants to use V4 signatures for GetObject, so most of the operations in https://github.com/awslabs/aws-java-sample work without the override.

At one point in stepping through and trying things that did not work (eg, a custom Region loaded from a resource file with <SignatureVersionOverride>2</SignatureVersionOverride>), I thought SignerFactory.registerSigner("S3SignerType", S3Signer.class); was also necessary, but it seems to work without that.

jamshid commented Sep 3, 2015

So to sum up, the way to make aws-sdk-jar 1.10.15 work with an S3-compatible service that does not implement V4 signatures is to:

        ClientConfiguration opts = new ClientConfiguration();
        opts.setSignerOverride("S3SignerType");  // NOT "AWS3SignerType"
        AmazonS3Client s3 = new AmazonS3Client(opts);
        s3.setEndpoint("https://storage.example.com");

It's tricky because as mentioned earlier, the SDK really wants to use V4 signatures for GetObject, so most of the operations in https://github.com/awslabs/aws-java-sample work without the override.

At one point in stepping through and trying things that did not work (eg, a custom Region loaded from a resource file with <SignatureVersionOverride>2</SignatureVersionOverride>), I thought SignerFactory.registerSigner("S3SignerType", S3Signer.class); was also necessary, but it seems to work without that.

@wangboak

This comment has been minimized.

Show comment
Hide comment
@wangboak

wangboak May 27, 2016

Hi,i don't understand why calculateStringToSignV2 use default 'POST',not use request.getMethod()? at com.amazonaws.auth.QueryStringSigner.

Hi,i don't understand why calculateStringToSignV2 use default 'POST',not use request.getMethod()? at com.amazonaws.auth.QueryStringSigner.

asfgit pushed a commit to apache/zeppelin that referenced this issue Oct 14, 2017

[ZEPPELIN-2950] Support Ceph as a notebook storage
### What is this PR for?
Make Zeppelin support Ceph as a notebook storage.

Ceph has APIs which are compatible with AWS S3 APIs. However, it supports only AWS Signature V2 and GetObject requests of aws-sdk-java use V4 by default: aws/aws-sdk-java#372

According to aws/aws-sdk-java#372 (comment) , the Zeppelin configuration of `zeppelin.notebook.s3.signerOverride` is added to make the `signerOverride` field of a `ClientConfiguration` instance configurable.

### What type of PR is it?
[Improvement]

### Todos

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2950

### How should this be tested?
Tested manually.

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: Keiji Yoshida <kjmrknsn@gmail.com>

Closes #2598 from kjmrknsn/feature/ceph-support and squashes the following commits:

c66cb63 [Keiji Yoshida] Support Ceph as a notebook storage
@wuseal

This comment has been minimized.

Show comment
Hide comment
@wuseal

wuseal Jul 18, 2018

Hi, Everyone, When setting this

        ClientConfiguration opts = new ClientConfiguration();
        opts.setSignerOverride("S3SignerType");  // NOT "AWS3SignerType"
        AmazonS3Client s3 = new AmazonS3Client(getCredProvider(context.getApplicationContext()),opts);

standpoint, It will work ok when upload file below 5 MB ,and when upload file bigger then 5MB .It will throw an exception like this:

com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: AFBCE47F09747899), S3 Extended Request ID: b2LJLc/1GjyflgfrUk90XdPjEZIKUR8EYt8EGHvP/qeBWDXpZAH9atPR2UGb6i/kw+y2wxz1NHI=
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:730)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:405)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:212)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4654)
        at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3485)
        at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.initiateMultipartUpload(UploadTask.java:270)
        at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.uploadMultipartAndWaitForCompletion(UploadTask.java:103)
        at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.call(UploadTask.java:83)
        at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.call(UploadTask.java:47)
        at java.util.concurrent.FutureTask.run(FutureTask.java:237)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
        at java.lang.Thread.run(Thread.java:761)

Help! Is there anything i missed?

wuseal commented Jul 18, 2018

Hi, Everyone, When setting this

        ClientConfiguration opts = new ClientConfiguration();
        opts.setSignerOverride("S3SignerType");  // NOT "AWS3SignerType"
        AmazonS3Client s3 = new AmazonS3Client(getCredProvider(context.getApplicationContext()),opts);

standpoint, It will work ok when upload file below 5 MB ,and when upload file bigger then 5MB .It will throw an exception like this:

com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: AFBCE47F09747899), S3 Extended Request ID: b2LJLc/1GjyflgfrUk90XdPjEZIKUR8EYt8EGHvP/qeBWDXpZAH9atPR2UGb6i/kw+y2wxz1NHI=
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:730)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:405)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:212)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4654)
        at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3485)
        at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.initiateMultipartUpload(UploadTask.java:270)
        at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.uploadMultipartAndWaitForCompletion(UploadTask.java:103)
        at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.call(UploadTask.java:83)
        at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.call(UploadTask.java:47)
        at java.util.concurrent.FutureTask.run(FutureTask.java:237)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
        at java.lang.Thread.run(Thread.java:761)

Help! Is there anything i missed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment