non amazonaws endpoint support (+ force PATH_STYLE)

[blajoie]

Hi,

Using the new v1.4 (current master)

It looks like when both ~/.s3cfg and ~/.aws credentials exist, then ~/.aws credentials will be used and ~/.s3cfg will be ignored.

We (@NHJohnson, @egor-dolzhenko) have a need to use the new host_base value extracted from the ~/.s3cfg (for non amazonaws endpoint), but we also do not want to force users to remove/disable ~/.aws credentials (this looks to be the only way to force htslib to always read (host_base) from ~/.s3cfg?)

This PR makes a few changes to hfile_s3.c.

always read from both ~/.s3cfg and ~/.aws/credentials, placing priority on ~/.aws for id/secret/token. This allows host_base to always be extracted and used. We are using a local S3 and specifying a non-amazonaws endpoint is required.

Following the logic outlined here (s3tools/s3cmd#416), we also have the need to always force PATH_STYLE over VIRTUAL_STYLE urls. Our s3 resource does not support VIRTUAL_STYLE urls. To achieve this, a user can set host_base and host_bucket to be equal, which then forces PATH_STYLE urls within htslib.

Any feedback is welcomed!

Thanks

[jmarshall]

This is the wrong approach. Mixing configuration settings from the two config files will rapidly become incomprehensible. The correct approach is to give up waiting for aws-cli upstream and define an equivalent of host_base in ~/.aws/credentials ourselves — this is #436.

OTOH a way to force path-style v. virtual-style is a good idea.

[blajoie]

Agreed completely - Glad we are done waiting for the upstream awscli fix :)

I would much rather define the equivalent of a host_base in the ~/.aws/credentials, perhaps even a flag within ~/.aws/credentials to force_path_style, url_mode=path/virtual?

Ideally host_base/url_mode would be set per profile, not global as it appears to be within ~/.s3cfg.

Thanks!

[jmarshall]

Likewise agreed. We've tried hard to use existing keys in ~/.aws/credentials and ~/.s3cfg (perhaps too hard in the case of endpoint_url!), so will look at what existing facilities to (ab)use for this path/virtual forcing.

HTSlib supports profiles in both these config files. Just put the settings in a [fooprofile] section instead of [default] and specify the profile to use in one of the usual ways. See recent comment on #436 for more details.

[blajoie]

Closing this as the better solution is to instead add a new param to ~/.aws/credentials
#436