AWS S3: Virtual host style requests fails witch custom endpoint (Minio)

[WojciechMazur]

Version 2.0.M3

When using virtual-host-style (which is now enabled by default) resulting endpoint does not include bucket name.
Expected request URI: http://some-bucket-name.minio-domain/some-key
Actual request URI: http://minio-domain/some-key

Bug is probably created in akka.stream.alpakka.s3.impl.HttpRequests requestAuthority function.
With current logic if endpoint is specified in config (when using minio it is always specified) then host is not enriched by bucket sub-domain.

Also it would be better if logging about deprecation would appear only 1 instead of logging in each usage.

[seglo]

Hi @WojciechMazur. Can you provide some more context about what API you're calling and the configuration/parameters. There's a test in HttpRequestsSpec that asserts the request is using hostname style.

https://github.com/akka/alpakka/blob/master/s3/src/test/scala/akka/stream/alpakka/s3/impl/HttpRequestsSpec.scala#L81

[WojciechMazur]

I've prepared spec which would fail in this example.
In my project I'm not using AWS S3, but Minio inside Kubernetes cluster. In this case I'm providing as endpoint for S3 node/service address.

    it should "create checkIfExits bucket request with custom endpoint" in {
    implicit val settings: S3Settings = getSettings().withEndpointUrl("http://custom.s3.storage:9000")

    val request: HttpRequest = HttpRequests.bucketManagementRequest(location, method = HttpMethods.HEAD)

    //Date is added by akka by default
    request.uri.authority.host.toString should equal("bucket.custom.s3.storage")
    request.uri.authority.port shouldEqual 9000
    request.entity.contentLengthOption should equal(Some(0))
    request.uri.queryString() should equal(None)
    request.method should equal(HttpMethods.HEAD)
  }

[seglo]

Yes, when the endpoint URL is explicitly defined it will use it as-is.

https://github.com/akka/alpakka/blob/master/s3/src/main/scala/akka/stream/alpakka/s3/impl/HttpRequests.scala#L181

If you encode the bucket into the endpoint URL itself everything seems to ok.

  it should "create checkIfExits bucket request with custom endpoint" in {
    implicit val settings: S3Settings = getSettings().withEndpointUrl("http://bucket.custom.s3.storage:9000")

    val request: HttpRequest = HttpRequests.bucketManagementRequest(location, method = HttpMethods.HEAD)

    //Date is added by akka by default
    request.uri.authority.host.toString should equal("bucket.custom.s3.storage")
    request.uri.authority.port shouldEqual 9000
    request.entity.contentLengthOption should equal(Some(0))
    request.uri.queryString() should equal(None)
    request.method should equal(HttpMethods.HEAD)
    request.uri.path.toString shouldEqual "/image-1024@2x"
  }

[ennru]

You're right, when an endpoint URL is specified, the bucket name will not be encoded into it which currently makes virtual-host-style access incompatible with it.

The endpoint URL should instead be given as some kind of format so that the bucket name could be added at the correct position. @WojciechMazur would you be in a position to suggest something like that in a pull request?

This partly relates to #1558.

[ennru]

I've looked into this, expect a PR today!

[ennru]

Fixed with #2193, but I've seen reports on Gitter that there might be parts missing.