Add S3 IPv6 "dualstack" endpoint support . Option = "use_dual_stack" [bool] #7482

tonymet · 2023-12-02T23:23:35Z

What is the purpose of this change?

Currently rclone hangs when IPv4 route to S3 is unavailable (e.g. no IPV4 interface or no IPV4 route to internet ). This change adds S3 IPv6 "dualstack" endpoint support . Option = use_dual_stack [bool] . Default S3 endpoints are IPv4 NOT IPv6

See AWS docs on IPv6:
"To make a request to an S3 bucket over IPv6, you need to use a dual-stack endpoint."

Why Now?

Starting Feb, EC2 will charge for public IPv4 addresses & NAT traffic. More AWS users will be using rclone & S3 on IPV6

Steps to Reproduce

Run test from a host with IPv6 address.
If the host has IPv4 address, use security-group, routing table or firewall to block IPv4 traffic (e.g. outbound destination 0.0.0.0 = block)
configure new rclone endpoint "test-s3"
run rclone ls test-s3:

ACTUAL RESULTS
rclone hangs

EXPECTED RESULTS
Files listed are visible

go run . ls s3-test:|head -1
502256548864 tonym-backup/2021-12-29/archive2.tar

Testing This Change

Checkout this commit
set use_dual_stack=true in rclone.conf (see below)
Set up your rclone machine to block IPv4 (see STEPS TO REPRODUCE)
go run . ls test-s3:

grep dual_stack~/.config/rclone/rclone.conf 
use_dual_stack= true

Testing S3 Default & DualStack endpoints

Example S3 queries to default & dual-stack endpoints for example

$ dig +short AAA s3.us-west-2.amazonaws.com|head -1
52.92.234.56
$ dig +short AAAA s3.us-west-2.amazonaws.com|head -1
# no answer

Dualstack (both IPv4 & IPv6 lookup work)

# 
 dig +short AAAA s3.dualstack.us-west-2.amazonaws.com |head -1
2600:1fa0:403f:aa89:345c:c3b8::
$ dig +short A s3.dualstack.us-west-2.amazonaws.com |head -1
52.92.243.184

Was the change discussed in an issue or in the forum before?

https://forum.rclone.org/t/ipv6-guide-review-with-google-drive-ec2-s3/43218

IPv6 Support Could Use Attention...

✅--bind to IPv6
❌oauth using IPv6 (::1)
(this PR) ❌Connect to S3 endpoint on IPv6
✅ Connect to Google Drive API endpoint on IPv6 (once authorized )
❓Connect to other backend endpoints on IPv6

I recommend dumping all rclone endpoints and testing dig AAAA to see which ones currently support IPv6.

Unit Testing

I've added test coverage s3.go covering enable/disable dualstack config

 cd backend/s3 ; go test .
ok      github.com/rclone/rclone/backend/s3     (cached)

no broken tests

 go test ./...

ok      github.com/rclone/rclone/backend/alias  0.023s
ok      github.com/rclone/rclone/backend/azureblob      0.024s
ok      github.com/rclone/rclone/backend/azurefiles     0.032s
# truncated

Checklist

[✅ ] I have read the contribution guidelines.
[✅ ] I have added tests for all changes in this PR if appropriate.
[✅ ] I have added documentation for the changes if appropriate.
[✅ ] All commit messages are in house style.
[✅ ] I'm done, this Pull Request is ready for review :-)

miyurusankalpa · 2023-12-03T15:43:59Z

Why not set this option enabled by default? Endpoint is dual stacked, so it should work on IPv4 only setups too.

tonymet · 2023-12-03T17:07:39Z

Why not set this option enabled by default? Endpoint is dual stacked, so it should work on IPv4 only setups too.

I'm following AWS cli pattern here. aws cli requires use_dualstack_endpoint = true to activate dualstack endpoints.

ncw

Thanks for doing this.

I had no idea that S3 endpoints wouldn't be dual stack by default. Seems rather quaint that they aren't but I guess AWS have their reasons.

I wrote some notes inline for some improvements.

Thank you :-)

ncw · 2023-12-04T10:30:39Z

backend/s3/s3.go

@@ -2628,6 +2628,7 @@ type Options struct {
 	Region                string               `config:"region"`
 	Endpoint              string               `config:"endpoint"`
 	STSEndpoint           string               `config:"sts_endpoint"`
+	DualstackEndpoint     bool                 `config:"dualstack_endpoint"`


This needs a config setting like this one

rclone/backend/s3/s3.go

Lines 2212 to 2221 in 298c13e

}, {

Name: "v2_auth",

Help: `If true use v2 authentication.

If this is false (the default) then rclone will use v4 authentication.

If it is set then rclone will use v2 authentication.

Use this only if v4 signatures don't work, e.g. pre Jewel/v10 CEPH.`,

Default: false,

Advanced: true,

ncw · 2023-12-04T10:31:05Z

docs/content/s3.md

@@ -1126,6 +1126,19 @@ Properties:
        - Owner gets FULL_CONTROL.
        - The AuthenticatedUsers group gets READ access.

+#### --s3-dualstack-endpoint


This should be auto generated from the config you need to add above :-)

ncw · 2023-12-04T10:37:39Z

backend/s3/s3.go

@@ -2956,6 +2957,9 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (*s3.S
 		r.addService("sts", opt.STSEndpoint)
 		awsConfig.WithEndpointResolver(r)
 	}
+	if opt.DualstackEndpoint {
+		awsConfig.WithUseDualStack(true)


The AWS docs say

// Deprecated: This option will continue to function for S3 and S3 Control for backwards compatibility. // UseDualStackEndpoint should be used to enable usage of a service's dual-stack endpoint for all service clients // moving forward. For S3 and S3 Control, when UseDualStackEndpoint is set to a non-zero value it takes higher // precedence then this option. UseDualStack *bool

About this option.

ncw · 2023-12-04T10:38:45Z

backend/s3/s3.go

@@ -2628,6 +2628,7 @@ type Options struct {
 	Region                string               `config:"region"`
 	Endpoint              string               `config:"endpoint"`
 	STSEndpoint           string               `config:"sts_endpoint"`
+	DualstackEndpoint     bool                 `config:"dualstack_endpoint"`


Can we call this use_dual_stack since it is a bool? That would fit in better with the other options.

Can we call this use_dual_stack since it is a bool? That would fit in better with the other options.

Thanks I’ll update with the feedback and thanks for catching the deprecation. There’s a v1 & v2 AWS api for GO and a couple other deprecations so this is one i missed. I’ll have the updates shortly.

Migrate to the new AWS SDK is probably something we should bite the bullet and do at some point, however that is out of scope for this PR :-)

tonymet · 2023-12-04T19:15:56Z

@ncw

✅ change to use_dual_stack
✅ use newer UseDualstackEndpoint API
✅ clean up error message in s3_test.go
✅ add doc config and cleanup .md file
Ready to merge

dualstack_endpoint=true enables IPv6 DNS lookup for S3 endpoints in s3.go, add Options.DualstackEndpoint to support IPv6 on S3

ncw

This looks great now - thanks!

I'll just run the CI and if it passed (modulo the thing I know will fail!) I'll merge - thank you :-)

ncw · 2023-12-08T11:11:57Z

Thank you :-)

tonymet · 2023-12-08T16:27:42Z

many thanks for all the help and guidance. I'm really happy to contribute to rclone it's been a great tool

tonymet force-pushed the s3-ipv6-dualstack branch 4 times, most recently from 088e7d4 to cf0ec77 Compare December 3, 2023 00:26

ncw reviewed Dec 4, 2023

View reviewed changes

tonymet force-pushed the s3-ipv6-dualstack branch 2 times, most recently from 15e9e76 to d45e387 Compare December 4, 2023 19:15

s3: S3 IPv6 support with option "use_dual_stack" (bool)

074b4ab

dualstack_endpoint=true enables IPv6 DNS lookup for S3 endpoints in s3.go, add Options.DualstackEndpoint to support IPv6 on S3

tonymet force-pushed the s3-ipv6-dualstack branch from d45e387 to 074b4ab Compare December 4, 2023 19:17

tonymet changed the title ~~Add S3 IPv6 "dualstack" endpoint support . Option = "dualstack_endpoint [bool]"~~ Add S3 IPv6 "dualstack" endpoint support . Option = "use_dual_stack" [bool] Dec 4, 2023

ncw approved these changes Dec 7, 2023

View reviewed changes

ncw merged commit 9fe343b into rclone:master Dec 8, 2023
9 of 10 checks passed

tonymet mentioned this pull request Jan 2, 2024

oauthutil: support IPv6 auth server listen bind address #7537

Draft

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add S3 IPv6 "dualstack" endpoint support . Option = "use_dual_stack" [bool] #7482

Add S3 IPv6 "dualstack" endpoint support . Option = "use_dual_stack" [bool] #7482

tonymet commented Dec 2, 2023 •

edited

miyurusankalpa commented Dec 3, 2023

tonymet commented Dec 3, 2023

ncw left a comment

ncw Dec 4, 2023

ncw Dec 4, 2023

ncw Dec 4, 2023

ncw Dec 4, 2023

tonymet Dec 4, 2023

ncw Dec 4, 2023

tonymet commented Dec 4, 2023

ncw left a comment

ncw commented Dec 8, 2023

tonymet commented Dec 8, 2023

	}, {
	Name: "v2_auth",
	Help: `If true use v2 authentication.

	If this is false (the default) then rclone will use v4 authentication.
	If it is set then rclone will use v2 authentication.

	Use this only if v4 signatures don't work, e.g. pre Jewel/v10 CEPH.`,
	Default: false,
	Advanced: true,

Add S3 IPv6 "dualstack" endpoint support . Option = "use_dual_stack" [bool] #7482

Add S3 IPv6 "dualstack" endpoint support . Option = "use_dual_stack" [bool] #7482

Conversation

tonymet commented Dec 2, 2023 • edited

What is the purpose of this change?

Why Now?

Steps to Reproduce

Testing This Change

Testing S3 Default & DualStack endpoints

Was the change discussed in an issue or in the forum before?

Unit Testing

Checklist

miyurusankalpa commented Dec 3, 2023

tonymet commented Dec 3, 2023

ncw left a comment

Choose a reason for hiding this comment

ncw Dec 4, 2023

Choose a reason for hiding this comment

ncw Dec 4, 2023

Choose a reason for hiding this comment

ncw Dec 4, 2023

Choose a reason for hiding this comment

ncw Dec 4, 2023

Choose a reason for hiding this comment

tonymet Dec 4, 2023

Choose a reason for hiding this comment

ncw Dec 4, 2023

Choose a reason for hiding this comment

tonymet commented Dec 4, 2023

ncw left a comment

Choose a reason for hiding this comment

ncw commented Dec 8, 2023

tonymet commented Dec 8, 2023

tonymet commented Dec 2, 2023 •

edited