Deploy to Ceph-based S3 gateway

[mfrancis95]

Hi. I'm trying to deploy to a Ceph-based S3 gateway.

Let's say my endpoint is https://myendpoint.com and the bucket is mybucket. The deployment tries to access mybucket.myendpoint.com:443 and gets a Failed to open TCP connection error because that name doesn't exist. The gateway is not set up to give buckets their own subdomains. Is there any way I can configure the deployment process to upload to just myendpoint.com:443?

The actual way in which a bucket is accessed using this gateway would be https://myendpoint.com/mybucket, not https://mybucket.myendpoint.com.

[svenfuchs]

@mfrancis95 thanks for the report!

I am thinking of adding an option subdomain that would default to true, but if set to false would make the provider choose https://[endpoint]/[bucket] as an upload target.

What do you think?

[mfrancis95]

That sounds great. I need the ability to choose just https://[endpoint] as the upload target though, unless what you said does essentially the same thing.

[svenfuchs]

I need the ability to choose just https://[endpoint] as the upload target though

Oh, ok? I think I'm confused. How does it then know which bucket the upload should go to?

In your original issue description you said:

The actual way in which a bucket is accessed using this gateway would be https://myendpoint.com/mybucket

... which made me thought you need https://[endpoint]/[bucket], not https://[endpoint].

Could you clarify? Is this just one single large bucket? How does it get to know which bucket you are referring to?

[mfrancis95]

The bucket variable is still being used. I think what you said is right.

I'm just coming from my limited experience using boto3 with a Ceph S3 gateway.

[svenfuchs]

I think what we are looking for here might be the option disable_host_prefix_injection? https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html

[svenfuchs]

@mfrancis95 i have no idea if this is what you are looking for, but i have added #1083 ... would you like to try this out?

[svenfuchs]

gonna merge the PR, closing this. feel free to reopen if needed.