S3 compatible need to support virtual hosted style(oss/nos)

[xiaolei0125]

current rclone s3 compatible type only supported path style, not virtual hosted style.
so rclone can not work with aliyun oss or netease cos.

eg:
path style: oss-cn-shenzhen.aliyuncs.com:/mybucket/myfile
virtual hosted style: mybucket.oss-cn-shenzhen.aliyuncs.com/myfile

rclone config
[oss]
type = s3
provider = Other
env_auth = false
access_key_id = LTAIyxxxxxxx
secret_access_key = plbDdxvBIuxxxxxxx
region =
endpoint = oss-cn-shenzhen.aliyuncs.com
location_constraint =
acl = public-read

rclone.exe --dump headers --log-level DEBUG lsd oss:mybucket
2018/07/16 18:14:40 DEBUG : GET /mybucket?delimiter=%2F&max-keys=1000&prefix= HTTP/1.1
Host: oss-cn-shenzhen.aliyuncs.com
User-Agent: rclone/v1.42
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20180716T101440Z
Accept-Encoding: gzip

2018/07/16 18:14:52 DEBUG : HTTP RESPONSE (req 0xc04235f800)
2018/07/16 18:14:52 DEBUG : HTTP/1.1 403 Forbidden
Content-Length: 260
Connection: keep-alive
Content-Type: application/xml
Date: Mon, 16 Jul 2018 10:14:53 GMT
Server: AliyunOSS
X-Amz-Request-Id: 5B4C701DB79FD65569BE0437

2018/07/16 18:14:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/16 18:14:52 ERROR : : error listing: SecondLevelDomainForbidden: Please use virtual hosted style to access.
status code: 403, request id: 5B4C701DB79FD65569BE0437, host id:
2018/07/16 18:14:52 Failed to lsd: SecondLevelDomainForbidden: Please use virtual hosted style to access.
status code: 403, request id: 5B4C701DB79FD65569BE0437, host id:

[ncw]

Rclone uses the aws official SDK. Can you work out what rclone would need to do in the SDK: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/ in order to enable this?

The code is here if you want to look at it: https://github.com/ncw/rclone/blob/master/backend/s3/s3.go

[xiaolei0125]

@ncw Thanks for you reply.
in https://github.com/ncw/rclone/blob/master/backend/s3/s3.go code:
awsConfig := aws.NewConfig().
WithRegion(opt.Region).
WithMaxRetries(maxRetries).
WithCredentials(cred).
WithEndpoint(opt.Endpoint).
WithHTTPClient(fshttp.NewClient(fs.Config)).
WithS3ForcePathStyle(true)

I think "WithS3ForcePathStyle(true)" should be modified to allow user to configure options， such as true or false,
WithS3ForcePathStyle(false) can support many cloud object storage service, such as aliyun oss and netease nos which only support virtual hosted style.

[ncw]

Ah, OK!

Try this with --s3-force-path-style=false and let me know if it fixes the problem.

You can also set this in the config file in the advanced section

https://beta.rclone.org/branch/v1.42-052-g56eeebc4-fix-2401-s3-path-style/ (uploaded in 15-30 mins)

If that works, would you document how to set up rclone with aliyun and netease?

[xiaolei0125]

@ncw Thank you for adding this new features.

I have tested it for aliyun oss and netease nos on windows and Linux, with --s3-force-path-style=false or advanced section, everything works fine.

so rclone with "--s3-force-path-style=false" can support some cloud object storage providers, such as aliyun oss and netease nos which compatible S3 API but not support path style.

[xiaolei0125]

This is a document show how to set up rclone with aliyun oss (netease nos has a similar configuration, only endpoint and access key is different) , see the attach files.
config_rclone_with_aliyun_oss.txt

[ncw]

Thank you for testing.

I've merged this to master and I've edited your doc slightly and merged that too in 01af8e2 - thank you very much indeed :-)

https://beta.rclone.org/v1.42-055-g01af8e29/ (uploaded in 15-30 mins)

This will be in the latest beta and v1.43