Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Presigned URL Generators #203

Open
sorenbs opened this issue Oct 10, 2017 · 26 comments

Comments

@sorenbs
Copy link

commented Oct 10, 2017

I can't find a way to generate a presigned url for s3. Is this feature missing or am I missing something?

@dagnir

This comment has been minimized.

Copy link
Contributor

commented Oct 10, 2017

@sorenbs You're correct, we don't currently have support for this in 2.0.

@dagnir

This comment has been minimized.

Copy link
Contributor

commented Oct 10, 2017

Related to #181

@aaronanderson

This comment has been minimized.

Copy link

commented Nov 3, 2017

While we wait for an official implementation here is a workaround I am successfully using. Basically I plugged in a custom ExecutionInterceptor to create a presigned URL right before the request goes over the wire and then I added a mock SdkHttpClient that doesn't actually execute the request.

@cesartl

This comment has been minimized.

Copy link

commented Feb 7, 2019

This is a significant hindrance to switching to version 2 of the API.
Is there any update on this?

@millems

This comment has been minimized.

Copy link
Contributor

commented Feb 7, 2019

It's on our roadmap, but we're unfortunately not able to comment on when it might be done. Bringing 2.x into parity with 1.11.x is a very high priority for our team.

@miere

This comment has been minimized.

Copy link

commented Jun 3, 2019

Hello maintainers,

It has been quite a long time since this issue have been opened. Do you have any thoughts on this? Are you willing to accept a PR for this feature? I mean, let me know if you have any plans to do so in the next couple of weeks, otherwise, I'd be glad to submit a PR.

I think this is a quite important feature and I'm afraid that, just as me, other developers would agree the S3 Java SDK 2.0 can't be considered finished without it.

@congtq

This comment has been minimized.

Copy link

commented Jun 13, 2019

It is almost 2 years since this issue was created. Do AWS really want developers to use 2.0?
To sad that now I have to switch back to 1.x after wasting days on implementation for AWS 2.0.

@cesartl

This comment has been minimized.

Copy link

commented Jun 13, 2019

I have to agree this is very strange indeed. Seems like a feature abandoned halfway through.

@debora-ito

This comment has been minimized.

Copy link
Collaborator

commented Jun 14, 2019

Hi everyone, just a quick update letting you know that we haven't forgotten this, it is in our V2 backlog along with other 1.11.x-parity features.

In the meantime, you can use the feature in SDK 1.x side by side with 2.x, here's an example on how to configure the pom.xml:
https://docs.aws.amazon.com/sdk-for-java/v2/migration-guide/side-by-side.html

@miere

This comment has been minimized.

Copy link

commented Jun 24, 2019

Hello @debora-ito. Thanks for you feedback and for your suggestion.

Unfortunately, I couldn't disagree more. Although I'm glad to finally hear something from AWS team regarding this issue, it's completely unacceptable to introduce even more overhead and bigger footprint to our deployment artifact by putting the both versions of the SDK at the same time.

I'm still open to make a PR and patch that for you, although the last time I've sent AWS a PR it took months to receive the first feedback on it.

If you ask me, I think such behavior is utterly inexcusable as we're a paying users willing to use more AWS IaaS/PaaS. Being completely honest with you, my report to my customers regarding AWS have changed quite a lot as other public cloud platforms have started to offer similar solutions and they have better customer support.

@albogdano

This comment has been minimized.

Copy link

commented Jun 24, 2019

I was just switching some of my code over to v2 and was shocked to find out this features is missing. This should be top priority for everyone involved in the development of SDK v2. I mean, come on guys, S3 is one of the core services! This is the second missing feature I encountered today. Earlier I was pretty disappointed to learn that the Waiter classes are missing as well. 👎

@acroz

This comment has been minimized.

Copy link

commented Jun 25, 2019

I strongly agree with other commenters. I have wasted a considerable amount of development time updating code to use the new SDK only to discover that core functionality is missing.

There is no indication anywhere in the documentation that the client is incomplete; in fact this page specifically says the contrary. There should at least be some strong warning that the S3 client is incomplete so that developers do not waste their time and money upgrading.

@millems

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

Sorry about the delay in responding, we wanted to come up with a response that we hope is more satisfying than “we promise we’re working on it”.

We agree with the sentiment from @acroz that we could do a better job communicating that this feature was missing. The documentation he linked says “all the service APIs are available today” which is very easy to interpret as “all methods on the 1.11.x client are available today” instead of our intended “all model-generated APIs which make remote service calls are available today”.

We’ve been working on bringing back the non-generated 1.11.x features to 2.x since GA, but we could have done a better job making it clear what our priorities are in that process.

What we’ll be doing based on this feedback:

  1. We’re going to move the pre-signers up in our backlog. It’s clear that depending on 1.11.x for just this feature is a bigger blocker than we thought.
  2. We’ll provide a code sample for you to use in the meantime, until the pre-signer task is completed. Unfortunately this isn’t something we can take a pull request on, because there are a number of services that need pre-signed URL support and we want our official SDK solution to support them all. (edit: link)
  3. We’re going to publicly publish our prioritized list of 1.11.x features that we are going to bring to 2.x. This will make it clear what we’re currently working on and what we are going to be working on next in this area. (edit: link)
  4. We’re going to link each feature in the backlog to an issue, and we will adjust the backlog’s order over time based on github feedback (upvotes, comments) and other similar channels.
  5. We’ll update the documentation @acroz linked to make it clear that presigners aren’t in 2.x of the SDK yet.
@cesartl

This comment has been minimized.

Copy link

commented Jun 26, 2019

Thanks @millems , what counts as upvote for this issue? Upvotes on the original comment?

@albogdano

This comment has been minimized.

Copy link

commented Jun 26, 2019

@cesartl Most likely it's the number of reactions to the original issue post. This is in fact the number 1 issue based on the number of reactions:
image

@millems

This comment has been minimized.

Copy link
Contributor

commented Jun 26, 2019

@cesartl @albogdano correct!

@millems

This comment has been minimized.

Copy link
Contributor

commented Jun 26, 2019

Below is a workaround you can use to generate a presigned S3 URL until the official SDK support is added. You will need to wait until tomorrow's release if you want to specify a custom expiration time... There's a bug we fixed today with presigned URLs to get it to work: #1310

URI uri = presign(PresignUrlRequest.builder()
                                   .region(Region.US_WEST_2)
                                   .bucket("millems-203")
                                   .key("test")
                                   .build());

public URI presign(PresignUrlRequest request) {
    String encodedBucket, encodedKey;
    try {
        encodedBucket = URLEncoder.encode(request.bucket(), "UTF-8");
        encodedKey = URLEncoder.encode(request.key(), "UTF-8");
    } catch (UnsupportedEncodingException e) {
        throw new UncheckedIOException(e);
    }

    SdkHttpFullRequest httpRequest =
            SdkHttpFullRequest.builder()
                              .method(request.httpMethod())
                              .protocol("https")
                              .host("s3-" + request.region().id() + ".amazonaws.com")
                              .encodedPath(encodedBucket + "/" + encodedKey)
                              .build();

    Instant expirationTime = request.signatureDuration() == null ? null : Instant.now().plus(request.signatureDuration());
    Aws4PresignerParams presignRequest =
            Aws4PresignerParams.builder()
                               .expirationTime(expirationTime)
                               .awsCredentials(request.credentialsProvider().resolveCredentials())
                               .signingName(S3Client.SERVICE_NAME)
                               .signingRegion(request.region())
                               .build();

    return AwsS3V4Signer.create().presign(httpRequest, presignRequest).getUri();
}

public static class PresignUrlRequest implements ToCopyableBuilder<PresignUrlRequest.Builder, PresignUrlRequest> {
    private final AwsCredentialsProvider credentialsProvider;
    private final SdkHttpMethod httpMethod;
    private final Region region;
    private final String bucket;
    private final String key;
    private final Duration signatureDuration;

    private PresignUrlRequest(Builder builder) {
        this.credentialsProvider = Validate.notNull(builder.credentialsProvider, "credentialsProvider");
        this.httpMethod = Validate.notNull(builder.httpMethod, "httpMethod");
        this.region = Validate.notNull(builder.region, "region");
        this.bucket = Validate.notNull(builder.bucket, "bucket");
        this.key = Validate.notNull(builder.key, "key");
        this.signatureDuration = builder.signatureDuration;
    }

    public static Builder builder() {
        return new Builder();
    }

    public AwsCredentialsProvider credentialsProvider() {
        return credentialsProvider;
    }

    public SdkHttpMethod httpMethod() {
        return httpMethod;
    }

    public Region region() {
        return region;
    }

    public String bucket() {
        return bucket;
    }

    public String key() {
        return key;
    }

    public Duration signatureDuration() {
        return signatureDuration;
    }

    @Override
    public Builder toBuilder() {
        return builder()
                .credentialsProvider(credentialsProvider)
                .region(region)
                .bucket(bucket)
                .key(key)
                .signatureDuration(signatureDuration);
    }

    public static class Builder implements CopyableBuilder<Builder, PresignUrlRequest> {
        private AwsCredentialsProvider credentialsProvider = DefaultCredentialsProvider.create();
        private SdkHttpMethod httpMethod = SdkHttpMethod.GET;
        private Region region;
        private String bucket;
        private String key;
        private Duration signatureDuration;

        public Builder credentialsProvider(AwsCredentialsProvider credentialsProvider) {
            this.credentialsProvider = credentialsProvider;
            return this;
        }

        public Builder httpMethod(SdkHttpMethod httpMethod) {
            this.httpMethod = httpMethod;
            return this;
        }

        public Builder region(Region region) {
            this.region = region;
            return this;
        }

        public Builder bucket(String bucket) {
            this.bucket = bucket;
            return this;
        }

        public Builder key(String key) {
            this.key = key;
            return this;
        }

        public Builder signatureDuration(Duration signatureDuration) {
            this.signatureDuration = signatureDuration;
            return this;
        }

        @Override
        public PresignUrlRequest build() {
            return new PresignUrlRequest(this);
        }
    }
}
@millems millems referenced this issue Jul 8, 2019

@millems millems changed the title Support GeneratePresignedUrlRequest Presigned URL Generators Jul 8, 2019

@millems millems added this to Backlog (Not Ordered) in New Features (Public) via automation Jul 8, 2019

@millems millems moved this from Backlog (Not Ordered) to Backlog (Ordered) in New Features (Public) Jul 8, 2019

@millems

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2019

We’re going to publicly publish our prioritized list of 1.11.x features that we are going to bring to 2.x. This will make it clear what we’re currently working on and what we are going to be working on next in this area.

We've published our backlog of new (public) features: https://github.com/aws/aws-sdk-java-v2/projects/1

Presigned URL generation is on top of the backlog at this time.

You can also keep an eye on what else we're working on and some of what we're planning to work on, minus any secret stuff that we can't talk about yet.

@jvinding

This comment has been minimized.

Copy link

commented Jul 26, 2019

                              .host("s3-" + request.region().id() + ".amazonaws.com")

There is no DNS entry for s3-us-east-1.amazonaws.com at the moment, so this code generating invalid URLs in that region.
is this a problem with the DNS, with the code, or are pre-signed URLs not supported in us-east-1?

@spfink

This comment has been minimized.

Copy link
Contributor

commented Jul 26, 2019

@jvinding not sure why the dash pattern was included in the code sample provided above. Replace "s3-" with "s3." and that should work in all regions including us-east-1.

@jvinding

This comment has been minimized.

Copy link

commented Jul 29, 2019

@jvinding not sure why the dash pattern was included in the code sample provided above. Replace "s3-" with "s3." and that should work in all regions including us-east-1.

Thanks for the response, unfortunately, that also fails with "The request signature we calculated does not match the signature you provided. Check your key and signing method."

I believe that the signer is not using the url in the request, but rather calculates the url based on the service and region using a -.

So, if I use . in the signing request, it won't actually match the url that gets signed.

@jvinding

This comment has been minimized.

Copy link

commented Aug 1, 2019

so, it seems there were 2 issues with the provided code, one is that s3-us-east-1.amazonaws.com does not exist.

I solved this by creating the url as so:

        val region = US_EAST_1 == request.region() ? "" : ("-" + request.region().id());
        final val httpRequest = SdkHttpFullRequest.builder()
                                                  .method(request.httpMethod())
                                                  .protocol("https")
                                                  .host("s3" + region + ".amazonaws.com")
                                                  .encodedPath(encodedBucket + "/" + encodedKey)
                                                  .build();

The other being that / in keys should not be encoded. e.g. a key of bob/joe/steve.png should remain untouched, but the example code turned it to bob%2fjoe%2fsteve.png.
I've fixed it by altering the encoding section of presign function as follows:

            encodedBucket = URLEncoder.encode(request.bucket(), "UTF-8");
            encodedKey = Arrays.stream(request.key().split("/"))
                               .map(s -> {
                                   try {
                                       return URLEncoder.encode(s, "UTF-8");
                                   } catch (UnsupportedEncodingException e) {
                                       throw new UncheckedIOException(e);
                                   }
                               })
                               .collect(Collectors.joining("/"));
@sigpwned

This comment has been minimized.

Copy link

commented Aug 25, 2019

I noticed that the sample code does not contain some important features for generating presigned URLs from the v1 library, namely the ability to provide custom Content-Type and Content-Disposition headers on the response. I've added these features to the code sample and integrated the feedback from others in the thread. I can confirm this code works for me in some unit tests I've written:

public class PresignUrlRequest
        implements ToCopyableBuilder<
            PresignUrlRequest.Builder,
            PresignUrlRequest> {
    public static URI presign(PresignUrlRequest request) {
        String encodedBucket, encodedKey;
        try {
            encodedBucket = URLEncoder.encode(request.bucket(), "UTF-8");
            
            encodedKey = Arrays.stream(request.key().split("/"))
                .map(s -> {
                    try {
                        return URLEncoder.encode(s, "UTF-8");
                    }
                    catch (UnsupportedEncodingException e) {
                        throw new UncheckedIOException(e);
                    }
                })
                .collect(joining("/"));
        }
        catch (UnsupportedEncodingException e) {
            throw new UncheckedIOException(e);
        }

        String regionPart=request.region() == Region.US_EAST_1
            ? ""
            : ("-" + request.region().id());
        SdkHttpFullRequest.Builder httpRequestBuilder =
            SdkHttpFullRequest.builder()
                .method(request.httpMethod())
                .protocol("https")
                .host("s3" + regionPart + ".amazonaws.com")
                .encodedPath(encodedBucket + "/" + encodedKey);
        
        if(request.responseContentDisposition() != null)
            httpRequestBuilder.appendRawQueryParameter(
                "response-content-disposition",
                request.responseContentDisposition());
        
        if(request.responseContentType() != null)
            httpRequestBuilder.appendRawQueryParameter(
                "response-content-type",
                request.responseContentType());
        
        SdkHttpFullRequest httpRequest=httpRequestBuilder.build();

        Instant expirationTime = request.signatureDuration() == null
            ? null
            : Instant.now().plus(request.signatureDuration());
        
        Aws4PresignerParams presignRequest =
            Aws4PresignerParams.builder()
                .expirationTime(expirationTime)
                .awsCredentials(request
                    .credentialsProvider()
                    .resolveCredentials())
                .signingName(S3Client.SERVICE_NAME)
                .signingRegion(request.region())
                .build();

        return AwsS3V4Signer.create()
            .presign(httpRequest, presignRequest)
            .getUri();
    }
    
    private final AwsCredentialsProvider credentialsProvider;
    private final SdkHttpMethod httpMethod;
    private final Region region;
    private final String bucket;
    private final String key;
    private final Duration signatureDuration;
    private final String responseContentType;
    private final String responseContentDisposition;

    private PresignUrlRequest(Builder builder) {
        this.credentialsProvider = Validate.notNull(builder.credentialsProvider, "credentialsProvider");
        this.httpMethod = Validate.notNull(builder.httpMethod, "httpMethod");
        this.region = Validate.notNull(builder.region, "region");
        this.bucket = Validate.notNull(builder.bucket, "bucket");
        this.key = Validate.notNull(builder.key, "key");
        this.signatureDuration = builder.signatureDuration;
        this.responseContentType = builder.responseContentType;
        this.responseContentDisposition = builder.responseContentDisposition;
    }

    public static Builder builder() {
        return new Builder();
    }

    public AwsCredentialsProvider credentialsProvider() {
        return credentialsProvider;
    }

    public SdkHttpMethod httpMethod() {
        return httpMethod;
    }

    public Region region() {
        return region;
    }

    public String bucket() {
        return bucket;
    }

    public String key() {
        return key;
    }

    public Duration signatureDuration() {
        return signatureDuration;
    }
    
    public String responseContentType() {
        return responseContentType;
    }
    
    public String responseContentDisposition() {
        return responseContentDisposition;
    }

    @Override
    public Builder toBuilder() {
        return builder()
            .credentialsProvider(credentialsProvider)
            .region(region)
            .bucket(bucket)
            .key(key)
            .signatureDuration(signatureDuration);
    }
    
    public static class Builder implements CopyableBuilder<Builder, PresignUrlRequest> {
        private AwsCredentialsProvider credentialsProvider = DefaultCredentialsProvider.create();
        private SdkHttpMethod httpMethod = SdkHttpMethod.GET;
        private Region region;
        private String bucket;
        private String key;
        private Duration signatureDuration;
        private String responseContentType;
        private String responseContentDisposition;

        public Builder credentialsProvider(AwsCredentialsProvider credentialsProvider) {
            this.credentialsProvider = credentialsProvider;
            return this;
        }

        public Builder httpMethod(SdkHttpMethod httpMethod) {
            this.httpMethod = httpMethod;
            return this;
        }

        public Builder region(Region region) {
            this.region = region;
            return this;
        }

        public Builder bucket(String bucket) {
            this.bucket = bucket;
            return this;
        }

        public Builder key(String key) {
            this.key = key;
            return this;
        }

        public Builder signatureDuration(Duration signatureDuration) {
            this.signatureDuration = signatureDuration;
            return this;
        }
        
        public Builder responseContentType(String responseContentType) {
            this.responseContentType = responseContentType;
            return this;
        }
        
        public Builder responseContentDisposition(String responseContentDisposition) {
            this.responseContentDisposition = responseContentDisposition;
            return this;
        }

        @Override
        public PresignUrlRequest build() {
            return new PresignUrlRequest(this);
        }
    }
}

All credit to @millems for the original sample code, and @jvinding for pointing out a couple of errors in it.

@mblund

This comment has been minimized.

Copy link

commented Sep 11, 2019

I had some problems with keys that contained spaces. My solution was to change the way the key was encoded and also set that the signer should not double encoded the url.

These are my changes if it helps someone else:

                encodedKey = Arrays.stream(request.key().split("/"))
                    .map(s -> {
                        try {
                            return URLEncoder.encode(s, "UTF-8")
                                    .replaceAll("\\+", "%20")
                                    .replaceAll("\\%21", "!")
                                    .replaceAll("\\%27", "'")
                                    .replaceAll("\\%28", "(")
                                    .replaceAll("\\%29", ")")
                                    .replaceAll("\\%7E", "~");
                        }
                        catch (UnsupportedEncodingException e) {
                            throw new UncheckedIOException(e);
                        }
                    })
                    .collect(joining("/"));
        Aws4PresignerParams params =
                Aws4PresignerParams.builder()
                        .expirationTime(expirationTime)
                        .awsCredentials(request
                                .credentialsProvider()
                                .resolveCredentials())
                        .signingName(SERVICE_NAME)
                        .signingRegion(request.region())
                        .doubleUrlEncode(false)
                        .build();
@Crysis21

This comment has been minimized.

Copy link

commented Sep 14, 2019

Wow, still no support for this feature. Why bother publish the version 2 of the SDK years ago, if you can't maintain it? Going back to 1.xx

@bendem

This comment has been minimized.

Copy link

commented Sep 17, 2019

I'd like to provided a simpler updated version (without builder and with APIs that don't require handling exceptions that never happen, thanks java 11). I've not tested it extensively with special characters but #203 (comment) shows a way to deal with that.

public URI generateUrl(AwsCredentialsProvider credentialsProvider, Region region, String bucket, String key) {
    var encodedBucket = URLEncoder.encode(bucket, StandardCharsets.UTF_8);
    var encodedKey = Stream.of(key.split("/"))
        .map(part -> URLEncoder.encode(part, StandardCharsets.UTF_8))
        .collect(Collectors.joining("/"));
    var host = "s3." + region.id() + ".amazonaws.com";

    var httpRequest = SdkHttpFullRequest.builder()
        .method(SdkHttpMethod.PUT)
        .protocol("https")
        .host(host)
        .encodedPath(encodedBucket + "/" + encodedKey)
        .build();

    var presignRequest = Aws4PresignerParams.builder()
        .awsCredentials(credentialsProvider.resolveCredentials())
        .signingName(S3Client.SERVICE_NAME)
        .signingRegion(region)
        .expirationTime(Instant.now().plus(5, ChronoUnit.MINUTES))
        .doubleUrlEncode(false)
        .build();

    return AwsS3V4Signer.create()
        .presign(httpRequest, presignRequest)
        .getUri();
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.