[aws_s3] Path to Virtual Hosted-Style workaround with middleware

[josephalfredo-okta]

I checked #485 and #253 (comment)

It looks like Virtual Hosted-Style is already on the roadmap, is there a recommended workaround for the current limitation?
S3 fips bucket is limited to virtual hosted style https://aws.amazon.com/compliance/fips/
I have been trying to figure out if we can do it manually with aws-smithy client middleware ourselves but it seems to erased the config passed, with endpoint and other credentials missing.

any guidance is appreciated

example code

    let shared_config = aws_config::load_from_env().await;
    let conf = aws_sdk_s3::config::Builder::from(&shared_config)
      .endpoint_resolver(
          Endpoint::immutable("http://localhost:8000".parse().expect("valid URI"))
      ).region(Region::new("us-east-1")).build();

    let smithy_client = aws_sdk_s3::client::Builder::dyn_https()
    .middleware_fn(|req: aws_smithy_http::operation::Request| {
     // the idea is to do the URI edit with existing data manually according to
     // https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/ 
      req 
    }).build_dyn();

    let client: Client = Client::with_config(smithy_client, conf);

result

[2022-10-07T20:23:17Z INFO  aws_smithy_http_tower::parse_response] send_operation; operation="HeadBucket"
[2022-10-07T20:23:17Z INFO  aws_smithy_http_tower::parse_response] send_operation; service="s3"
[2022-10-07T20:23:17Z DEBUG hyper::client::client] Client requires absolute-form URIs, received: /mybucket1665174197299
[2022-10-07T20:23:17Z INFO  aws_smithy_http_tower::parse_response] send_operation; status="dispatch_failure"
[2022-10-07T20:23:17Z INFO  aws_smithy_http_tower::parse_response] send_operation; message=user error: client requires absolute-form URIs

Thank you!

[jdisanti]

Customizing the middleware on the underlying Smithy client replaces the entire middleware stack, so the built-in credential and endpoint middlewares (among others) from the SDK won't be in the chain unless they're also added during client creation.

Fortunately, there's a better way to customize requests now that doesn't require touching middleware. For example:

let response = client.head_bucket()
    .bucket("example")
    .customize()
    .await?
    .mutate_request(|mut req| {
        // change the URI on `req` here
    })
    .send()
    .await?;

[josephalfredo-okta]

This seems to only allow modification on the relative url, is there a way to also modify the endpoint? since the vhost style put the put the bucket name in the front
https://bucket-name.s3.region-code.amazonaws.com/key-name

[jdisanti]

Ah, my bad. I thought that would just work, but actually, the endpoint middleware runs after the operation customization, and thus, replaces the endpoint that was customized.

What you could do in the meantime is create a S3 client just for this bucket, and set its endpoint resolver to the correct endpoint. Then keep the operation customization to remove the bucket from the URI path.

None of this is ideal. Keep an eye out for the Endpoints 2.0 issue that's being worked on. It will deliver built-in support for virtual hosted-style buckets.

[zhangkaizhao]

Thanks you very much for this workaround! I have used it to successfully resolve the issue of using the awesome aws-sdk-s3 crate with Aliyun OSS which only supports virtual hosted-style. 👍

[martijnarts]

Here's how I ended up generating a signed URL with vhost style:

    let request_uri = format!("/{key}", key = request.key);
    let mut request =
        rusoto_core::signature::SignedRequest::new("GET", "s3", &Region::UsEast1, &request_uri);
    request.set_hostname(Some(format!("{}.{}", bucket_name, endpoint)));

    let non_cdn_uri =
        request.generate_presigned_url(&aws_credentials, &Duration::from_secs(5 * 60), false);
    let uri = http::Uri::try_from(&non_cdn_uri)?;

    let mut parts = uri.into_parts();
    *parts.authority.as_mut().unwrap() =
        http::uri::Authority::try_from(config.cdn_hostname.clone())?;
    let new_uri: http::Uri = parts.try_into()?;

    Ok(new_uri.to_string())

My Rust is not the best yet, so I might've messed up some things here but it works!

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.