Support for DNS style S3 Bucket naming

[adevress]

Simple question regarding Bucket addressing convention and minio 👍

AWS S3 defines two way to address a bucket : http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html

• Path style. With something like http://server.adress/%(bucket)

• Name / DNS style. With something like http://%(bucket).server.adress/

Is Minio supporting the DNS / Name style ? I do not find any documentation related to that.
If not, do you plan to support it in future ?

Adev

[vadmeste]

@adevress, Minio doesn't support DNS style. The reason behind that is that we want Minio to be simple with the least possible knobs. Any reason why you need DNS style ?

[adevress]

@vadmeste Some of the tools I would like to use, including some proprietary one support only DNS style naming unfortunatly.

To my understanding, DNS style naming should not have a big cost in term of complexity for minio. Adding a simple parameter name of the style "--server-name=mys3.mydomain.com" would allow you to resolve any query of the style "mybucket.mys3.mydomain.com" by looking into the HTTP Host header no ?

[krishnasrinivas]

@adevress all the softwares/client-libs we have seen till now support a "path-style" config which would send requests as path-style instead of virtual host style. I suspect your software might also allow such a config, can you check? Would you mind sharing the name of the software? We can check too.

[adevress]

The software itself: objectiveFS ( https://objectivefs.com/ )

log structure filesystem constructed on a S3 backend.

[krishnasrinivas]

objectiveFS does not support path-style:
https://objectivefs.com/howto/how-to-use-ofs-with-on-premise-object-store

@adevress we'll discuss about how to go about this and let you know.

[adevress]

@krishnasrinivas @vadmeste thanks guys

[krishnasrinivas]

@adevress for now we are not going to support virtual-host-style. All AWS-SDKs and most S3-clients we have seen support path-style. It would be easy for ObjectiveFS to support path-style if they use AWS-SDKs.

[adevress]

@krishnasrinivas I understand your decision but this is regrettable. All other major S3 implementations including Amazon itself, RadosGW and Pithos support DNS Bucket styl. Path style addressing start to be marked as deprecated in several toolkit including the official AWS Java toolkit itself.

I think minio should consider to implement it in future, I can contribute to that if needed.

[krishnasrinivas]

@adevress let me check if we can accept your contribution if you implement it.

[deekoder]

@adevress - we will have a discussion and update this thread. There are a few moving parts to consider before taking on this feature and doing it right. So give us some more time to get back to you about this.

[nlowe]

Would it be theoretically possible to do this at a reverse proxy (a la nginx)?

[rmb938]

You can do it with nginx but the s3 protocol doesn't like url rewrites.

[adevress]

Would it be theoretically possible to do this at a reverse proxy

Using a proxy or doing any path rewrite cause issue with the S3 auth system.

[harshavardhana]

DNS style bucket naming requires manual intervention on the part of user, IMO this defeats the simplicity we are aiming for with Minio. Minio is simple enough to deploy and forget with no more additional DNS entries to be made, /etc/hosts to managed (NOTE: which all will require root access on the host system). Not just that each bucket needs to be resolvable to the host system through DNS and present on the host system. (NOTE: Remove bucket requires removal of entries from the DNS as well).

It is possible for Minio to implement this feature but at the cost of deployment complexity which is the main hindrance of not doing it, in-fact we had this working 2yrs+ ago - but we choose the path towards simplicity and not complicate Minio deployments.

AWS S3 supports DNS style bucket naming because they choose to make bucket globally unique which is also a sort of a hindrance on users part that we would have to choose a unique bucketName and cannot use simple communicable names anymore.

Please chime in your views here @adevress @rmb938 @nlowe

[rmb938]

@harshavardhana I do agree in keeping Minio simple, it is pretty easy to get up and running without any manual steps.

However for "public" buckets it is nice to have custom/vanity dns names that are user facing. i.e for static websites.

One idea would allow the bucket to have an optional setting for a dns name and it would be up to the deployer to put a nginx proxy with the correct settings infront as well as changes on whatever dns server. I personally don't have a current use case though, just thought I would provide some input.

Not really sure how hard that would be to implement or if it is even worth that added complexity.

[adevress]

DNS style bucket naming requires manual intervention on the part of user, IMO this defeats the simplicity we are aiming for with Minio.

The only real mandatory user intervention is to add a "wildcard" DNS entry to the minio server. This is travial to do with most DNS provider nowadays, and not more complicated than a "standard" A entry which need anyway to be done for any decent usage.

/etc/hosts to managed (NOTE: which all will require root access on the host system).

I don't think you need to manage /etc/hosts nor root access to simply get the server name. A simple config parameter will do the job.

One idea would allow the bucket to have an optional setting for a dns name and it would be up to the deployer to put a nginx proxy with the correct settings infront as well as changes on whatever dns server. I personally don't have a current use case though, just thought I would provide some input.

It is not possible to "cheat" with a nginx.
Up to my knowledge because it violate the authentication mechanism. The S3 authentication system works by hashing the hostname. This will violate the signature if the hostname change in case of DNS bucket style to path style conversion inside the proxy. Making the proxy re-compute the signature is non-trivial and potentially a security hole.

[harshavardhana]

The only real mandatory user intervention is to add a "wildcard" DNS entry to the minio server. This is travial to do with most DNS provider nowadays, and not more complicated than a "standard" A entry which need anyway to be done for any decent usage.

We could potentially leverage DNS wildcard here but we would need to provide this like an optionally configured behavior, only available when you have a fully qualified DNS provider and a resolvable DNS pointing back to the server where minio is running or proxy.

Additional problems you might have to know is that if you have a TLS requirement (which is probably the case here with publicly facing minio server) then bucket names such as bucket.name cannot be used directly due to TLS wildcard restrictions this applies to AWS S3 buckets as well. I do not know if the clients which only use DNS style can ever support buckets with such names.

I don't think you need to manage /etc/hosts nor root access to simply get the server name. A simple config parameter will do the job.

This i was thinking aloud when you do not have a DNS provider. From what i read it would mean that if you do have a fully qualified DNS and also a wildcard setup this feature can kick in.

It is not possible to "cheat" with a nginx.

This is correct assessment @adevress

I personally think this is a niche requirement, proprietary implementations not supporting path style is perhaps not a good reason to do this as i explained forcing DNS style can lead to TLS errors with incompatible bucket names. Knowing all these details in a deployment is not only hard and in-fact wrong to expect from our users.

With all this information internally we will asses the situation and take a call. Thanks for your inputs @adevress @rmb938

[adevress]

Additional problems you might have to know is that if you have a TLS requirement (which is probably the case here with publicly facing minio server) then bucket names such as bucket.name cannot be used directly due to TLS wildcard restrictions this applies to AWS S3 buckets as well. I do not know if the clients which only use DNS style can ever support buckets with such names.

It is for this reason that the official S3 has a rule for that
"When using virtual hosted–style buckets with SSL, the SSL wildcard certificate only matches buckets that do not contain periods. To work around this, use HTTP or write your own certificate verification logic. We recommend that you do not use periods (".") in bucket names. "

I personally think this is a niche requirement, proprietary implementations not supporting path style is perhaps not a good reason to do this as i explained forcing DNS style can lead to TLS errors with incompatible bucket names.

I personnally think you should not see it as a feature to support a proprietary implementation. Amazon S3 supports v4 signature and DNS-style-bucket everywhere now and already started to deprecate path-style-bucket and v2 signature, all of their newer sites simply do not support it at all.
It is deprecated and will very certainly disappear.
It is just a matter of time before mainstream S3 client start also to drop support for v2 signature and path style buckets, some of them already flat it as "deprecated".

This is going to impact minio if you do not support it.

[harshavardhana]

It is for this reason that the official S3 has a rule for that
"When using virtual hosted–style buckets with SSL, the SSL wildcard certificate only matches buckets that do not contain periods. To work around this, use HTTP or write your own certificate verification logic. We recommend that you do not use periods (".") in bucket names. "

Yes correct i do know why it doesn't work - you should look at makeTargetURL() code on how minio-go handles for all S3 endpoints and providers intelligently based on all the current restrictions. This is the whole reason to implement our own SDKs since AWS SDKs do a poor job at automatically managing these gritty details.

I personnally think you should not see it as a feature to support a proprietary implementation. Amazon S3 supports v4 signature and DNS-style-bucket everywhere now and already started to deprecate path-style-bucket and v2 signature, all of their newer sites simply do not support it at all.

This is an incorrect assessment i couldn't find anywhere that is listed as deprecated in their docs in-fact
it is clearly written - http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html

Amazon S3 supports virtual hosted-style and path-style access in all regions. The path-style syntax, however, requires that you use the region-specific endpoint when attempting to access a bucket. For example, if you have a bucket called mybucket that resides in the EU (Ireland) region, you want to use path-style syntax, and the object is named puppy.jpg, the correct URI is http://s3-eu-west-1.amazonaws.com/mybucket/puppy.jpg.

But since you personally vested in this feature we will consider this for discussion - think through all the details of deployment and usage. Thanks for chiming in for your side of the argument here @adevress

[harshavardhana]

We discussed this internally and agreed to proceed on implementing DNS style bucket names

• This feature is on only when server is started with --address <fqdn>

[adevress]

We discussed this internally and agreed to proceed on implementing DNS style bucket names

This feature is on only when server is started with --address <fqdn>

Awesome !

Thank you very much guys for your reactivity and your efficiency.

[jamshid]

I'm glad this feature was implemented but I'm curious...

@adevress you said Amazon "already started to deprecate path-style-bucket". Do you have a link about that? I can't find anything.

If anything it seems more S3 clients (e.g. ChronoSync) are supporting path style access, either making it the default or adding an option for it, because it makes using S3-compatible storage much easier.

[mappu]

Do you have a link about that? I can't find anything.

There was just an official announcement:

Announcement: Amazon S3 will no longer support path-style API requests starting September 30th, 2020.
-- https://forums.aws.amazon.com/ann.jspa?annID=6776

[dmolesUC]

The latest news seems to be that they'll continue to support path-style routing for older buckets, which sounds like good news as far as continued tool/SDK support for path-style routing. (A bit annoying for anyone trying to write tools that work with both, though.)

Original Plan – Support for the path-style model ends on September 30, 2020.

Revised Plan – Support for the path-style model continues for buckets created on or before September 30, 2020. Buckets created after that date must be referenced using the virtual-hosted model.

https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

[harshavardhana]

FWIW MinIO already supports DNS style buckets so either way, the AWS S3 plan seems fine either way.

[dmolesUC]

But not by default, right? And depending on what you're running Minio behind, there might be firewall / vhost / etc. issues with enabling it.

[harshavardhana]

What matters is there is a simple a way to enable it.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.