Support S3 interface endpoints (AWSPrivateLink for Amazon S3)

[cyberduck]

068bb52 created the issue

S3 interface endpoints enable to connect to Amazon S3 using a private IP address:

• AWS PrivateLink for Amazon S3

I am unable to connect to Amazon S3 using the interface endpoint URL (e.g. vpce-0971cacd1f2xxxxxxxxx.s3.eu-west-1.vpce.amazonaws.com) as the server hostname. Cyberduck continously tries to authenticate (I see thousands of packets in Wireshark) and fails a few minutes later. The issue comes from how Cyberduck generates the SigV4 signature, because it considers that "vpce" is the region (e.g. HTTP header Authorization is AWS4-HMAC-SHA256 Credential=AKIASFI36Y5VXXXXXXX/20210702/vpce/s3/aws4_request which fails).

I think that two things should be corrected in Cyberduck:

• Fetch the region differently from the server endpoint URL
• Consider S3 interface endpoint URL as "special URL" and use this URL only (don't use dualstack or North Virginia as the default region to list existing S3 buckets)

As a workaround, I was able to connect to a S3 interface endpoint by:

• Resolving s3.eu-west-1.amazonaws.com to one of the private IP addresses of the interface endpoint (added an entry in the hosts file)
• Applying the default parameters:

s3.bucket.virtualhost.disable=true
s3.endpoint.dualstack.enable=false
s3.endpoint.format.ipv4=s3.eu-west-1.amazonaws.com

However, it would be great if Cyberduck could natively support S3 interface endpoints, without all these tricks.

[cyberduck]

@dkocher commented

We already check the hostname using the regular expression ([a-z0-9\-]+\.)?s3(\.dualstack)?(\.[a-z0-9\-]+)?\.amazonaws\.com and thus no usage of the dual stack endpoints should apply.

[cyberduck]

@dkocher commented

Dependent on upstream changeset to detect region.

[cyberduck]

@dkocher commented

Existing documentation to disable the use of virtual host style requests in connection profile.

[cyberduck]

@dkocher commented

Regional DNS names include a unique VPC endpoint ID, a service identifier, the AWS Region, and vpce.amazonaws.com in its name. For example, for VPC endpoint ID vpce-1a2b3c4d, the DNS name generated might be similar to vpce-1a2b3c4d-5e6f.s3.us-east-1.vpce.amazonaws.com.

[cyberduck]

@dkocher commented

In 4ef31c5 with upstream fix. Please try the latest snapshot build with this connection profile.

[cyberduck]

068bb52 commented

Hello,

Many thanks for the quick update! I tested with Access Key and Secret and it works. However I tried to create a mix profile with STS credentials and Private Link, and it does not work. The profile is:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "_Apple_DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Protocol</key>
        <string>s3</string>
        <key>Vendor</key>
        <string>s3-sts-privatelink</string>
        <key>Scheme</key>
        <string>https</string>
        <key>Description</key>
        <string>S3 (STS and Private Link)</string>
        <key>Default Nickname</key>
        <string>S3 (STS and Private Link)</string>
        <key>Username Placeholder</key>
        <string>Profile Name in ~/.aws/credentials</string>
        <key>Password Configurable</key>
        <false/>
        <key>Token Configurable</key>
        <false/>
        <key>Anonymous Configurable</key>
        <false/>
        <key>Properties</key>
        <array>
            <string>s3service.disable-dns-buckets=true</string>
            <string>s3.endpoint.dualstack.enable=false</string>
        </array>
    </dict>
</plist>

It works when the server is s3.amazonaws.com, but I get an error message "The AWS Access Key Id you provided does not exist in our records". It seems like it does not use STS credentials.

Is that the expected behavior?
Thanks!

[cyberduck]

@dkocher commented

Replying to [comment:13 malaval]:

It works when the server is s3.amazonaws.com, but I get an error message "The AWS Access Key Id you provided does not exist in our records". It seems like it does not use STS credentials.

Is that the expected behavior?
Thanks!
This requires an additional fix as we currently handle the VPC endpoints as non AWS S3.

[cyberduck]

@dkocher commented

Additional fix in a910429.

[cyberduck]

068bb52 commented

Thank you! I don't manage to get the last release work. With the latest snapshot build (35295), I tried to both approaches and none of them worked for me. What should be the cyberduckprofile to use STS credentials with VPC endpoints?

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "_Apple_DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Protocol</key>
        <string>s3</string>
        <key>Vendor</key>
        <string>s3-privatelink</string>
        <key>Scheme</key>
        <string>https</string>
        <key>Description</key>
        <string>AWS PrivateLink for Amazon S3 (VPC endpoint)</string>
        <key>Hostname Configurable</key>
        <true/>
        <key>Port Configurable</key>
        <true/>
        <key>Username Placeholder</key>
        <string>Profile Name in ~/.aws/credentials</string>
        <key>Password Configurable</key>
        <false/>
        <key>Token Configurable</key>
        <false/>
        <key>Anonymous Configurable</key>
        <false/>
        <key>Properties</key>
        <array>
            <string>s3service.disable-dns-buckets=true</string>
            <string>s3.endpoint.dualstack.enable=false</string>
        </array>
    </dict>
</plist>

Or

<!DOCTYPE plist PUBLIC "_Apple_DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Protocol</key>
        <string>s3</string>
        <key>Vendor</key>
        <string>s3-token</string>
        <key>Description</key>
        <string>S3 (Credentials from AWS Security Token Service)</string>
        <key>Default Nickname</key>
        <string>S3 (Credentials from AWS Security Token Service)</string>
        <key>Username Placeholder</key>
        <string>Profile Name in ~/.aws/credentials</string>
        <key>Password Configurable</key>
        <false/>
        <key>Token Configurable</key>
        <false/>
        <key>Anonymous Configurable</key>
        <false/>
        <key>Properties</key>
        <array>
            <string>s3service.disable-dns-buckets=true</string>
            <string>s3.endpoint.dualstack.enable=false</string>
        </array>
    </dict>
</plist>

[cyberduck]

@dkocher commented

Can you please attach the debug log output for the connection attempt. To enable debug logging open a Terminal.app window and enter

defaults write ~/Library/Preferences/ch.sudo.cyberduck.plist logging debug

Log output can be found in the cyberduck.log file in~/Library/Logs/Cyberduck. You can easily reach this file in Console.app (Open from /Applications/Utilities) under Reports → Log Reports → cyberduck.log.